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PROJECT  SUMMARY  OVERVIEW 
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•  Overall  Goal:  understand  the  unique  cognitive  mechanisms  that  should  be  employed  to 

optimize  collaborative  decision-making  activity  in  a  geographically  distributed 
and  time-delayed  situation 


•  Objectives:  (1)  To  understand  the  cognitive  processes  of  building  knowledge  in  an 

asynchronous,  distributed  collaboration  environment 

(2)  To  develop  an  empirically-based  theory  of  collaboration,  including 
knowledge  building  components,  during  asynchronous,distributed 
collaboration 


(3)  To  understand  how  agents  can  support  humans  in  achieving  collaborative 
knowledge  during  asynchronous,  distributed  collaborative  problem  solving 


•  FY03  /  04  Objectives:  (1)  Conduct  experiments  in  collaborative  problem  solving  in  both 

face-to-face  and  asynchronous,  distributed  environments  to 
understand  the  unique  cognitive  processes  within  asynchronous, 
distributed  collaboration 


(2)  Update  model  of  collaboration  based  on  experimental  results 


EXPECTED  FINAL  PRODUCTS 
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1.  Empirical  Data  /  Journal  Articles 

•  Describing  the  cognitive  processes  of  building  knowledge  with  distributed 
team  members  engaged  in  asynchronous,  quick  response  collaboration 

2.  Empirically-based  Theory  for  Asynchronous,  Distributed  Collaboration  including 
Knowledge  Building  Components 

3.  Empirical  Data  on  Agent  Support  for  Achieving  Collaborative  Knowledge  in 
Asynchronous,  Distributed  Collaboration 

4.  Demonstration  of  the  Knowledge  Building  Processes  during  Asynchronous, 
Distributed  Collaborative  Problem  Solving  using  an  Integrated  EWall  /  NEO 
Scenario  Testbed. 

•  Testbed  is  laptop  based,  portable  and  can  be  used  to  demonstrate  Ewall 
collaboration  tool  and  knowledge  building  processes  to  other  agencies 

(e.g.  NPGS,  universities)  and  Operational  communities  (e.g.  PACOM,  JIFCOM, 
SPA  WAR  FORCEnet) 


Gr^tuiidUiiit 


Asynchronous,  Distributed 

Ewall  Collaboration  Tool  With  Agent  Based  Knowledge  Building  Model 


Network  Centric  Warfare 


lication 
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Improve  total  collaborative  problem  solving  time  of  an  Asynchronous,  Distributed  team  to 
that  of  a  Face-To-Face  team.  Applicable  across  different  domains 
(e.g.  Network  Centric  Warfare) 

*  Asynchronous,  Distributed  teams  are  usually  an  order  of  magnitude 
slower  than  Face-To-Face  teams  (i.e.  from  literature  and  CASC  data) 

*  Improvement  achieved  through: 

(1)  understanding  the  collaboration  stages  and  cognitive  processes  used  in 
Face-To-Face  and  Asynchronous,  Distributed  Collaborative  teams 

(2)  understand  the  collaboration  dynamics  between  team  members 
(i.e.  the  transition  states  for  both  stages  and  cognitive  processes) 

(3)  Using  knowledge  from  (1)  &  (2)  develop  an  agent-based  support  system 
for  Asynchronous,  Distributed  collaborative  teams 

Military  impact  —  optimize  collaborative  decision-making  activity  in  asynchronous, 
distributed  teams  for  mission  planning  and  mission  execution 


SCIENTIFIC  QUESTIONS  ADDRESSED 
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•  What  is  the  knowledge  building  process  humans  use  in  asynchronous,  distributed 
collaborative  problem  solving? 

•  What  is  the  effect  of  the  following  variables  on  the  knowledge  building  process 
during  asynchronous,  distributed  collaboration? 

*  Collaboration  Modes  (face-to-face  versus  asynchronous,  distributed) 

*  Knowledge  Distribution  (homogeneous  versus  heterogeneous) 

*  Knowledge  Dynamics  (static  knowledge  versus  dynamic  knowledge) 

•  What  forms  of  agent  support  facilitate  knowledge  building  in  asynchronous, 
distributed  collaborative  problem  solving? 


Contribution  to  Cognitive  Science 
and  Collaboration  Technology 
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Understand  the  knowledge  building  process 
humans  use  in  asynchronous,  distributed 
collaborative  problem  solving  and  compare  to 
existing  Face-To-Face  collaboration  literature 

Extend  existing  collaboration  theories  to  include 
knowledge  building  processes  for  asynchronous, 
distributed  collaborative  teams 


OVERVIEW  OF 

PHASE  I,  II,  AND  HI  EXPERIMENTS 
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•  Phase  I  Objective  =  collect  empirical  data  on  the  collaboration  stages  and  cognitive  processes 
during  asynchronous,  distributed  team  collaboration  in  a  collaborative  problem  solving  domain 
(murder  mystery  scenario).  Emphasis  is  on  the  the  cognitive  processes  of  knowledge  building. 

-  Examine  the  effects  of  collaboration  mode  (face-to-face  vs  asynchronous,  distributed), 
and  knowledge  distribution  (homogeneous  vs  heterogeneous) 


•  Phase  II  Objective  =  building  on  phase  I  examine  the  effects  of  knowledge  dynamics 
(static  vs  dynamic  knowledge)  on  the  cognitive  knowledge  building  process  using  the 
NEO  scenario  within  the  Ewall  collaboration  environment 


•  Phase  III  Objective  =  within  the  context  of  the  revised  theory  for  knowledge  building 
in  asynchronous,  distributed  collaboration  (based  on  Phase  I  and  II  results),  define  and 
empirically  examine  the  beneficial  forms  of  agent  support? 


CASC  Phase  I  Scenario 

Murder  Mystery  ^  j^y  ^  I  R 

See  Appendix  A  for  detail  murder  mystery  information  


The  Case  of  the  Fallen  Businessman,  ( by  Dr.  Garold  Stasser,  Miami  University,  Oxford,  Ohio) 


Major  Characters 

Robert  Gill: 
Mary  Gill: 

Lt.  Mark  Moody: 
Sgt.  Cassini: 
**Eddie  Sullivan: 
**Billy  Prentice: 


The  victim 
The  victim’s  wife 

Detective  in  charge  of  the  investigation 
Police  officer  assisting  in  the  investigation 
Handyman  who  worked  for  the  Gills 
Yardman  who  worked  for  the  Gills 


**Mickey  Malone:  Owner  of  MM  Auto  Parts;  business  associate 


Sam  Nietzel: 
Dave  Daniels: 


Parts  manager  for  Gill  Lincoln/Mercury 
Owner  of  Dave’s  Quick  Stop  in  the  Eastwood 


**  The  ONLY  suspects  under  consideration  are:  Mickey  Malone 

Billy  Prentice 
Eddie  Sullivan 


of  the  victim 
Shopping  Center 


Summary:  Robert  Gill,  a  prominent  local  businessman  was  found  dead  behind  his  Crestview  home 
this  morning.  Detective  Lt.  Mark  Moody  of  the  Hilltown  precinct  reported  that  Mr.  Gill  had 
apparently  been  assaulted  when  leaving  his  home  to  play  golf  early  this  morning.  He  was  struck 
on  the  head  over  the  left  eye  and  fell  down  a  flight  of  stairs  leading  from  a  second  story  deck  at  the 
rear  of  the  house.  The  preliminary  coroner’s  report  concluded  that  death  was  caused  by  injuries 
sustained  from  the  fall  and  not  from  the  blow  to  the  head.  The  report  estimated  that  Mr.  Gill’s 
death  occurred  between  6:30  and  7:00  AM.  Lt.  Moody  would  neither  confirm  nor  deny  rumors  that 
Mr.  Gill  had  been  robbed.  “We’re  following  all  leads.  That’s  all  I  have  to  say  for  now,”  said  Lt.  Moody. 

Team  Objective:  Collaborate  on  the  detailed  murder  information  and  develop  a  team  consensus 

on  who  killed  Mr.  Gill 


TECHNICAL  PLAN 
CASC  Phase  I  Experiment 
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Independent  Variables: 


-  Collaboration  Mode  (face-to-face  vs  asynchronous,  distributed) 

*  Face-to-Face  =  team  interacts  synchronously  with  each  other  through  speech 

*  Asynchronous,  Distributed  =  team  interacts  with  each  other  at  different  times 

and  from  different  locations  through  a  text  based 
web  forum 


-  Knowledse  Distribution  (homogeneous  vs  heterogeneous) 

*  Homogeneous  =  the  members  of  the  team  have  all  the  murder  mystery 

knowledge  in  common 

*  Heterogeneous  =  the  members  of  the  team  have  some  murder  mystery 

knowledge  in  common  and  some  uniquely  held  murder 
mystery  knowledge 


CASC  Phase  I  Experimental  Design 

(2x2  randomized  factorial) 
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Knowledge  Distribution 


Collaboration  Face-to-Face 
Mode 

Asynchronous, 

Distributed 

(text) 


Homogeneous 

Heterogeneous 

Gpl 

Gp  15 

* 

* 

* 

* 

Gp7 

Gp21 

Gp8 

Gp  22 

* 

* 

* 

* 

Gpl4 

Gp28 

Dependent  Variables: 


Phase  I 

•  28  groups  total 

•  3  subjects  /  group 

•  84  subjects  total 


-  Forum  text  and  face-to-face  audio  /  video  recordings  including  time  stamp  per 
response  (i.e.  text  and  speech) 


-  Thinking  Aloud  Protocol  (concurrent  verbalizations)  for  asynchronous,  distributed 
teams 


-  Total  time  to  successfully  complete  the  problem-solving  task  (time  from  the 
beginning  of  the  task  until  task  completion) 

-  Collaboration  Maps  (post  session  -  subjects  construct  a  map  of  their  view  of  the 
stages  &cognitive  process  states  of  team  collaboration) 

-  Subjective  Questionnaire  -  measuring  expertise,  trust  between  team  members,  and 
general  collaboration  opinions  among  members 


CASC  Phase  I  Hypotheses 
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Ho:  no  significant  difference  between  face-to-face  and  asynchronous,  distributed 

collaboration  modes  on  the  collaboration  stages  and  cognitive  knowledge  building 
processes  in  a  collaborative  problem  solving  domain 

Ho:  no  significant  difference  between  homogeneous  and  heterogeneous  knowledge 

distribution  on  the  collaboration  stages  and  cognitive  knowledge  building  processes 
in  a  collaborative  problem  solving  domain 

Ho:  no  significant  interaction  between  collaboration  modes  and  knowledge  distribution  on 
the  collaboration  stages  and  cognitive  knowledge  building  process 

Ho:  no  significant  difference  in  time  or  frequency  within  each  cognitive  process  state  across 
collaboration  mode  and  knowledge  distribution  conditions 

Ho:  no  convergence  of  individual  mental  models  with  regard  to  collaboration  stages  and 
knowledge  building  processes 


Types  of  Data  Analysis 
Phase  I,  II  and  III  Experiments 
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•  Verbal  Protocol  Communication  Analyses  -  identification  of  collaboration  stages  and  cognitive 
process  states  compared  across  collaboration  mode  and  knowledge  distribution.  Compare  results 
to  preliminary  structural  model  of  collaboration. 

•  Transition  State  Dia2rams  -  representation  of  the  dynamic  team  collaborative  behavior  between 
collaboration  stages  and  between  cognitive  process  states  within  each  collaboration  stage  compared 
across  collaboration  mode  and  knowledge  distribution.  Compare  results  to  preliminary 
structural  model  of  collaboration. 

•  Parametric  statistics  -  for  analyzing  time,  and  frequency  within  each  collaboration  stage  and 
cognitive  process  state  across  collaboration  mode  and  knowledge  distribution  conditions.  Also  used 
for  analyzing  total  time  to  complete  task  and  questionnaire  data. 

•  Collaboration  Maps  -  determine  the  degree  of  convergence  between  individual  mental  model ‘s 
regarding  collaboration  stages  and  cognitive  processes.  In  addition,  compare  how  an  individual 
thinks  a  group  makes  a  decision  in  a  collaborative  setting  and  how  the  group  actually  performs. 


CASC  Phase  I  Experiment 

Face-to-Face,  Homogeneous  Team  Collaboration 
fshows  card  sorting  approach  to  information  management 


Example  N  AV;^AIR 


CASC  Phase  I  Experiment 

Face-to-Face,  Homogeneous  Team  Collaboration 
(shows  using  maps  to  transfer  meaning  to  team  members) 


ExampIeNAVj^AIR 
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CASC  PHASE  I  EXPERIMENT 
Asynchronous,  Distributed,  Homogeneous  Team  Example 
Using  Forum  Collaboration  Environment 
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MODEL  EVOLUTION 
OF 

TEAM  COLLABORATION 
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INITIAL  THEORY  FOR  ACHIEVING 
COLLABORATIVE  KNOWLEDGE 
IN  ASYNCHRONOUS  COLLABORATION 


TEAMWISE  COLLABORATION  MODEL 
(Rawlings, 2000) 

'  interplay  between  interdependent  tasks  and  team 
>  req.  conditions  =  shared  purpose  /  vision,  efficient 
collaborative  work  /  c^ability  to  achieve  goals 


Time,  Interaction  and  Performance  (TIP) 
propositions  (McGrath,  1991): 

’  N  ature  of  groups  •  Temporal  factors 


Group  interaction  process 


KNOWLEDGE  BUILDING  PROCESS 


(Stahl,  2  00  0) 


=  FORM  S  OF 


C  ollaborative 
know  ledge 


^Negotiate  jperspectiv 


1 


Formalize  &  objectify 

Use  in  performing  task 

▼ 

Update  personal 
unders^nding 


shared 

understanding 


argum  entation 
&  rationale 

D  is  cuss  ^alternatives 


personal 
.understanding 


A  rticulate 


other  people’s 
statem  ents 
(  i.e.  Team) 


M  ETRIC  S 


C  ollaboration 

•  information  rich  group  discussions 

•  collective  decision  making 

•  integrated,  joint  solutions  derived 
from  group 

Knowledge  Building  Process 

•  Thinking  Aloud  Protocol  (verbalize) 

•  Concept  Map  (pre  /  post  session) 

•  Percentage  of  Domain  Relevant 
Information  exchanged  (verbal /text 
protocol  analyses) 

•  Number  of  Integrated,  Joint  Solutions 

•  Time  to  successfully  C0rtlplet6 

com  m  on  task 


Preliminary  Conceptual  Model  of  Collaboration 


ream  Qrientati^ 

•  getting  acquainted 

•  clarifying  task 
initial  attitudes 


Model 


Level  2 
(Processes) 


Individual 

Understanding 


Articulate  pto  Team 


Accumulation 
of  facts  (Team) 


I  I  =  Knowledge 
Building 


Update 


Team  Conflict 

•  decision  alternatives 

•  criticism  of  alternative^ 


•  Decision  Alternatives 

•  Criticism  of  Alternatives 

(with  rationale) 


Iteration  loop  for  selecting  decisi  on’ 


Emergence  of^ 
Team  Decisions 

•  decisions  emerge y 
from  tearn. 


Neg(  >tiate  Perspectives 


of  alternatives 


Collaborative 

Knowledge 


Team  Shared 
Understanding 


^einforcemem 
/of  Team  Decisionsl 

•  team  consensus 


STRUCTURAL  MODEL  OF  TEAM  COLLABORATION 


Problem  Area 
Characteristics 


Team 

Consensus 


Implement 
Decision  To  Solve 
Task 


i/ 


•  culturally  diverse 

•  heterogeneous  knowledge 

•  unique  roles 

•  command  structure 
(hierarchical  vs.  flat) 

•  rotating  team  members 


•  develop  shared  understanding 


Meta-Cognitive: 

•  individual  understanding  of  'i  *  goal  development  ; 

problem  conditions  ;  •  team  maital  model  of  problem  ; 

•  individual  mental  model  developmejit .  ^  \ 

of  situational  significance  i  j 


•  track  team's  mental  model  changes  '  i  •  compare  problem  solution 


•  problem  identification 

•  understanding  problem  task 

■  es^Wi^teamcommunicaticm 

■  establish  data  filtering  methods 

■  establish  meaning  transfer  conventions 


I  •  goal  definition  i 

I  •  iterative  information  collection  ; 


•Knowledge  Building: 

■  problem  definition 

■  individual  task  knowledge 

■  individual  team  knowledge 


•team  maital  model  of  team  j  •  collaborative  knowledge 

•team  task  knowledge  j  •  shared  understanding 

•  domain  ejqtertise 

•  shared  understanding 

•  collaborative  knowledge 

m  for  Information  Processing  and  Knowledge  Bnilding  (applies  to  all  stages): 


•  goal  requirements 


•  presenting  individual  information 

•  disagreement 

•  questioning 


•  discussing  individual  information 


•  discussion  of  possible  solutions 


•  discussing  team  generated  information 

•  providing  rationale  for  individual  solutions 

•  agreement 
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STRUCTURAL  MODEL  OF  TEAM  COLLABORATION 


Problem  Area 
Characteristics 


Collaborative  Situation 


Parameters: 


•  time  pressure 

•  information/knowledge 
uncertainty 

•  dynamic  information 

•  large  amount  of  knowledge 
(cognitive  overload) 

•  human-agent  interface 
complexity 

Team  Tvoes 


•  asynchronous 

•  distributed 

•  culturally  diverse 

•  heterogeneous  knowledge 

•  unique  roles 

•  command  structure 
(hierarchical  vs.  flat) 

•  rotating  team  members 
Operational  Tasks 


•  team  decision  making,  COA 
selection 

•  develop  shared  understanding 

•  intelligence  analysis 
(team  data  processing) 


Team 
Knowledge 
Base 

Construction 


v: 


Collaboration  Stases 


Collaborative 
Team  Problem 
Solving 


Team 
Consensus 


Outcome 
Evaluation 
and  Revision 


Collaboration 

Complete 


Yes 


'  Achieve^ 
K  Goal  j 


No 


Meta-Cognitive: 


•  individual  understanding  of 

problem  conditions 

•  individual  mental  model  developmei 
of  situational  significance 

Ittformatioja.  Pr.oc.e&sias; . 

•  goal  development 

•  team  mental  model  of  problem 

•  team  plan  to  solve  problem 

•  track  team's  mental  model  changes  ’  ■ 

•  understanding  remaining  items  to 
resolve 

•  compare  problem  solution  i 
against  goals 

•  problem  identification 

•  understanding  problem  task 

•  establish  team  communication 
and  trust 

•  establish  data  filtering  methods 

•  establish  meaning  transfer  conventiq 

•  goal  definition 

•  iterative  information  collection 
&  analysis 

•  develop,  rationalize,  &  visualize 
solution  alternatives 

jns 

•  team  negotiation  of  solution 
alternatives 

•  analyze,  revise  output 

•Knowledge  Building: 

•  problem  definition 

•  individual  task  knowledge 

•  individual  team  knowledge 

•  team  mental  model  of  team 

•  team  task  knowledge 

•  domain  expertise 

•  shared  understanding 

•  collaborative  knowledge 

•  collaborative  knowledge 

•  shared  understanding 

•  goal  requirements 

•  exit  criteria 

Communication  Mechanism  for  Information  Processing  and  Knowledge  Building  (applies  to  all  stages): 


*  presenting  individual  information 

*  disagreement 

*  questioning 


•  discussing  individual  information 

•  negotiating  perspectives 

•  discussion  of  possible  solutions 


•  discussing  team  generated  information 

•  providing  rationale  for  individual  solutions 

•  agreement 


CASC  Phase  I  Collaboration  Stages 
Coding  Definitions 

(see  Appendix  B  for  complete  list  of  examples) 
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•  Team  Knowled2e  Base  Construction  (TK):  “Team  members  reading,  clarifying  information  and 
understanding  the  problem.” 

*  Defining  the  problem  (realizing  what  they  have  to  accomplish  — 

“We  need  to  come  up  with  a  solution  in  30  minutes.”) 

*  Clarifying  the  facts  —  “What  did  they  say  about  Joe’s  hearing? 

•  Collaborative  Team  Problem  Solvin2  (TPS):  “Team  members  working  together  to  develop 
solution  options  to  the  problem.” 

*  Analyzing  the  facts  to  come  up  with  a  solution - “I  think  Joe  did  it  because  he  tried  to  blame 

everything  on  Tim.” 

•  Team  Consensus  (TC):  “Team  negotiation  of  solution  option  and  final  agreement  by  all  team 
members  on  a  particular  option.” 

*  Agreeing  on  the  final  solution  —  “Do  we  all  agree  that  Joe  did  it?” 

•  Outcome  Evaluation  &  Revision  (OER):  “Team  evaluation  of  selected  solution  option  against 
problem  solving  goal.  Team  revises  solution  option  if  option  does  not  meet  goal.” 

*  Choosing  to  accept  the  final  decision  or  revise  it - “I  know  we  all  agreed  on  Tim,  but  I  think 

we  better  take  a  closer  look  at  Joe.” 


•  Additional  Sta2es  (MISC):  “Other  unique  team  behavior  not  described  in  the  above  categories.” 
*  Need  to  describe  unique  behavior  and  label  stage. 


CASC  Phase  I  Cognitive  Process  States 
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(see  Appendix  B  for  complete  list  of  examples) 
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Team  Knowledge  Base  Construction;  (TK)  “Team  members  reading  clarifying  and  understanding  the  problem  to  be  solved.” 


IPup:  Information  Processing  (understanding  problem)  =  stating  the  facts  without  applying  that  knowledge  to  any  possible  solution. 
“Tim’s  muffler  was  loud” 

IPgd:  Information  Processing  (goal  definition)  =  defining  the  team  goal. 

“Our  goal  is  to  determine  who  killed  Mr.  Gill” 

IPct:  Information  Processing  (communication  &  trust)  =  establishing  team  trust. 

“I  understand  why  you  thought  it  was  Tim.” 

IPdfm:  Information  Processing  (data  filtering  methods)  =  team  uses  methods  to  sort  data. 

“using  yellow  note  pads  to  sort  data  into  categories” 

IPtck:  Information  Processing  (transfer  conventions)  =  team  uses  methods  to  transfer  meaning  to  other  team  members  during 

knowledge  construction. 

“team  uses  map  of  Mr.  Gill’s  house  to  understand  distances  between  objects. 

KBpd:  Knowledge  Building  (problem  definition)  =  defining  the  problem  (realizing  what  they  have  to  accomplish) 

“We  need  to  come  up  with  a  solution  in  30  minutes” 

KBik:  Knowledge  Building  (indiv.  knowledge)  =  individual  clarifying  the  facts;  asking  for  clarification. 

“What  did  they  say  about  Joe’s  hearing” 

KBtk:  Knowledge  Building  (team  knowledge)  =  team  clarifying  facts 
“Billy  left  the  coffee  shop  at  7:00am” 

“No  that  was  Eddie  that  left  the  coffee  shop  at  7:00am 
“Yes  that  was  Eddie” 


CASC  Phase  I  Cognitive  Process  States 
Coding  Definitions 

(see  Appendix  B  for  complete  list  of  examples) 
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Collaborative  Team  Problem  Solving;  (TPS)  “Team  members  working  together  to  develop  solution  options  to  the  problem.” 


MCtp:  Meta-cognitive  (team  plan)  =  establishing  a  plan  of  approaeh. 

“ADHES  2,  why  don’t  you  say  who  you  think  it  is,  then  I’ll  say  who  I  think  it  is.” 

IPica:  Information  Processing  (information  collection  &  analysis)  =  collecting  and  analyzing  the  facts  to  come  up  with  a  solution  but  no 

specific  solution  mentioned. 

“The  killer  seems  to  blame  everything  on  Tim.” 

IPtcp:  Information  Processing  (transfer  conventions)  =  team  uses  methods  to  transfer  meaning  to  other  team  members  during  team  problem 

solving. 

“  Using  maps  or  yellow  stickers  to  transfer  meaning  to  team  members” 

IPsa:  Information  Processing  (solution  alternatives)  =  developing,  rationalizing  and  discussing  solution  alternatives. 

“I  think  Eddie  did  it?” 

KBde:  Knowledge  Building  (domain  expertise)  =  team  members  state  their  own  domain  knowledge  and  apply  that  knowledge  to  the  problem. 

“Yes  I  am  a  Maryland  State  Trooper  and  they  is  no  concrete  evidence  for  Billy  killing  Mr.  Gill.” 

KBsu:  Knowledge  Building  (shared  understanding)  =  using  facts  to  justify  a  solution. 

“I  think  Eddie  did  it  because  he  was  hard  of  hearing  and  had  his  fingerprints  on  the  crowbar.” 

KBck:  Knowledge  Building  (collaborative  knowledge)  =  Convincing  others  of  a  specific  thought  without  absolute  consensus  of  the  final 

solution. 


“You’re  right.  I  didn’t  think  about  that.” 


CASC  Phase  I  Cognitive  Process  States 

Coding  Definitions 

(see  Appendix  B  for  complete  list  of  examples) 
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Team  Consensus;  (TC)  “Team  negotiation  of  solution  option  and  final  agreement  by  all  team  members  on  a  particular  option.” 


MCitr:  Meta-cognitive  (items  to  resolve)  =  team  stating  remaining  items  to  resolve. 

“We  need  to  compare  Mickey’s  and  Eddie’s  location  times  with  Billy’s” 

IPtn:  Information  Processing  (team  negotiation)  =  team  negotiation  of  solution  alternatives  ending  in  a  final  solution. 

“  I  believe  it  is  Eddie  because  he  was  hard  of  hearing” 

“  I  concur  which  also  means  Eddie  was  lying  about  where  is  was” 

“I  also  think  it  was  Eddie  because  of  his  fingerprints  on  the  crowbar. 

“We  all  in  consensus,  it  was  Eddie” 

KBckc:  Knowledge  Building  (collaborative  knowledge)  =  convincing  others  of  a  specific  thought  without  absolute  consensus  of  the  final 

solution. 

“You’re  right.  I  didn’t  think  about  that.” 

KBsuc:  Knowledge  Building  (shared  understanding)  =  using  facts  to  justify  a  solution. 

“It  couldn’t  have  been  Tim  because  his  muffler  was  loud  and  the  car  at  the  coffee  shop  had  a  quiet  muffler.” 


CASC  Phase  I  Cognitive  Process  States 
Coding  Definitions 
(see  Appendix  B  for  complete  list  of  examples) 
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Outcome  Evaluation  &  Revision:  (PER)  “Team  evaluation  of  selected  solution  option  against  problem  solving  goal.  Team  revises  solution 

option  if  option  does  not  meet  goal.” 

MCps:  Meta-cognitive  (problem  solution)  =  compare  problem  solution  against  goal(s). 

“Are  we  done  or  do  we  want  to  talk  about  it  some  more?” 

IPar:  Information  Processing  (analyze,  revise  output)  =  analyze  final  solution  option  and  revise  if  necessary. 

“I  know  we  all  agreed  on  Tim,  but  I  think  we’d  better  take  a  closer  look  at  Joe.” 

KBgo:  Knowledge  Building  (goal  obtainment)  =  understand  if  solution  option  fits  specific  goal  criteria. 

“  the  murder  had  a  truck  with  a  loud  muffler,  was  at  Mr.  Gills  house  around  10:30am,  and  pushed  Mr.  Gill  down  the  porch  steps”. 
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Mean  %  Time  by  Collaboration  Stages 

Collaboration  Mode  (CM)  vs.  Knowledge  Distribution  (KD) 


Team  Knowledge  Base  Construction: 

CM:  F=  5.51323,  p  =  0.027449 
KD:  F  =  170682,  p  =  0.203780 
CM*KD:  F=  6.02627,  p  =  0.021724 
N=84 


Collaborative  Team  Problem  Solving: 

CM:  F=  6.944,  p  =  0.014504 
KD:  F  =2.010,  p  =  0.169097 
CM*KD:  F  =  1876,  p  =  0.183471 
N=84 


Team  Consensus: 

CM:  F  =0.5-654,  p  =  0.478422 
KD:  F=  3.43855,  p  =  0.07602 
CM*KD:  F=  0.82967,  p  =  0.371425 
N=84 


Collaboration  Mode/Knowledge  Distribution: 


□  Face  to  Face  Homogeneous 

□  Asynchronous,  Distributed  Homogeneous 


I  Face  to  Face  Heterogeneous 
I  Asynchronous,  Distributed  Heterogeneous 
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Mean  %  Time  By  Cognitive  Process  States: 
Team  Knowledge  Construction  (TK) 

Wilks  lambda  =  0.402182,  F  =  3.158681,  p  =  0.022010 
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Mean  %  Time  By  Cognitive  Process  States: 
Coilaborative  Team  Problem  Solving  (TPS) 
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Mean  %  Time  By  Cognitive  Process  States: 
Team  Consensus  (TC) 
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Collaboration  Stages 


•  Face-to-Face  heterogeneous  teams  spend  significantly  more  time  in  team  knowledge  construction 
than  asynchronous,  distributed  heterogeneous  teams.  There  was  no  difference  between  the 
homogeneous  teams. 

•Asynchronous,  distributed  teams  spend  more  time  in  the  collaborative  problem  solving  stage  than 
face-to-face  teams.  There  was  no  difference  between  homogeneous  and  heterogeneous  teams  with 
regard  to  time  spent  in  the  collaborative  problem  solving  stage. 

•  There  was  no  difference  in  the  amount  of  time  spent  in  team  consensus  and  outcome  evaluation  & 
revision  stages  between  collaboration  mode  and  knowledge  distribution  conditions 


CASC  Phase  I  Conclusions 

Cognitive  Process  States 
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•  In  team  knowledge  base  construction  (TK)  the  face-to-face  teams  spend  significantly  more  time 
developing  individual  knowledge  and  team  knowledge  than  the  asynchronous,  distributed  teams. 
Asynchronous,  distributed  teams  spend  significantly  more  time  understanding  the  problem 
compared  to  Face-to-Face  teams. 

•  In  collaborative  team  problem  solving  (TPS)  the  face-to-face  teams  spend  significantly  more  time  in 
using  conventions  to  transfer  meaning  compared  to  asynchronous,  distributed  teams.  Asynchronous, 
distributed  teams  spend  significantly  more  time  developing  solution  alternatives  compared  to  Face- 
to-Face  teams. 

•  In  team  consensus  (TC)  there  were  no  significant  differences  between  Face-to-Face  and 
Asynchronous,  Distributed  teams  in  team  consensus  process  states. 


CASC  Phase  I  Conclusions 

Transition  Probabilities  and  Paths 
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•  Both  face-to-face  and  asynchronous,  distributed  teams  demonstrated  dynamic  transitions  between 
Team  Knowledge  Base  Construction  (TK),  Collaborative  Problem  Solving, (TPS)  and 
Team  Consensus  stages  (TC).  Outcome  Evaluation  &  Revision  stage  was  not  used  by  either  team 
type  because  there  was  no  need  to  revise  given  our  domain  task. 

*  Face-to-face  teams  demonstrated  mostly  a  linear  path  between  TK,  TPS  and  TC  whereas 
asynchronous,  distributed  teams  showed  non-linear  path  with  feedback  loops  (area  for 
agent  support) 


•  During  team  knowledge  base  construction(TK)  face-to-face  teams  demonstrated  more  transitions 
between  the  TK  cognitive  process  states  than  the  asynchronous,  distributed  teams.  Transition 
probabilities  for  the  face-to-face  teams  were  higher  in  most  cases  compared  to  the  asynchronous, 
distributed  teams 

•  During  collaborative  team  problem  solving  (TPS)  both  team  types  focused  on  the  information 
collection  and  analysis  process  state  (IPica).  Also,  the  IPica  state  was  the  main  transition  state  from 
the  TK  stage. 


CASC  Phase  I  Conclusions 

Transition  Probabilities  and  Paths  (continued) 
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•  During  team  consensus  both  team  types  showed  dynamic  transitions  between  the  process  states 
of  team  negotiation  and  shared  understanding.  Low  probability  of  transition  back  to  cognitive 
process  states  within  Team  Problem  Solving  or  Team  Knowledge  Construction  stages 

•No  dynamic  transitions  into  the  cognitive  process  states  of  the  Outcome  Evaluation  &  Revision 
stage  by  either  team  type 


Project  Status 
FY04  Plans 
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Phase  II  Experiment 


•  Collaboration  Task: 

>  Expanded  NEO  Mission  Scenario  (Warner,  Wroblewski,  Schuck,  Cowen,  Letsky,  2003) 

*  see  Appendix  C  for  detail  scenario 

*  developed  with  expertise  from  operational  personnel  (Navy  Seal,  Marine,  Army  aircrew) 

>  Mission  Statement  — 

The  time  is  2:00am,  January  15.  Your  mission  is  to  rescue  3  stranded  Red  Cross  workers  from  a  church  basement,  on 
a  remote  island,  caught  in  the  middle  of  guerilla  warfare,  within  24  hours.  The  situation  is  described  in  the  next  few 
pages  along  with  the  assets  of  US  forces  which  are  available  to  rescue  the  workers.  You  need  to  work  together  and 
develop  a  course  of  action  (using  ANY  assets  available  to  you),  which  includes  a  plan  for  getting  to  the  church,  a  plan 
for  evacuating  the  workers,  and  a  plan  for  the  return  to  the  Army  base  or  aircraft  carrier.  The  course  of  action  solution 
can  be  an  Army,  Marine,  Navy  Seals  solution,  or  a  combination  of  the  assets  of  the  three.  You  want  to  choose  the 
optimal  and  most  efficient  solution.  You  want  to  minimize  damage  to  the  village  and  villagers;  you  want  to  avoid  contact 
with  enemy  if  possible,  and  to  rescue  the  workers  safely.  However,  the  rules  of  engagement  are  that  any  forces  will 
defend  themselves  if  needed.  Good  Luck! 

•  Asynchronous^  Distributed  Collaboration  Environment; 

>  Modified  Ewall  workstation  and  visualization  area 

*  see  Appendix  D  for  design  specification  for  Ewall  modifications 

*  see  next  four  slides  for  modified  Ewall  workstation  display 


CASC  Client  Application  -  Station  1  Connected: 1 92. 1 68.0. 1 22 
File  Network  Help 
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Independent  Variables: 

-  Collaboration  Mode  (face-to-face  vs  asynchronous,  distributed) 

*  Face-to-Face  =  team  interacts  synchronously  with  each  other  through  speech 

*  Asynchronous,  Distributed  =  team  interacts  with  each  other  at  different  times 
and  from  different  locations  through  the  Ewall  collaboration  environment 

-  Knowled2e  Dynamics  (static  vs  dynamic  knowledge) 

*  static  knowledge  =  all  information  (I.e.  background  information,  rebels.  Navy 

Seals,  Marine  and  Army  assets,  Intell,  weapons  and 
environment  expertise  information)  remains  the  same 
throughout  the  collaborative  NEO  scenario  problem 

*  dynamic  knowledge  =  Selected  information  (I.e.  rebels,  weather,  time  to  rescue, 

condition  of  red  cross  workers)  changes  at  a  standard 
time  in  the  collaborative  NEO  scenario  problem 


CASC  PHASE  II  EXPERIMENTAL  DESIGN 

(2x2  randomized  factorial) 
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Knowled2e  Dynamics 
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Phase  II 

•  32  groups  total 

•  3  subjects  /  group 

•  96  subjects  total 


Dependent  Variables: 

Ewall  cards  and  face-to-face  audio  /  video  recordm2s  including  time  stamp  per  response 
(i.e.  cards  and  speech) 

-  Total  time  to  successfully  complete  the  problem-solving  task  (time  from  the  beginning  of 
the  task  until  task  completion) 

-  Collaboration  Maps  (post  session  -  subjects  construct  a  map  of  their  view  of  the  stages  & 
cognitive  process  states  of  team  collaboration) 


-  Subjective  Questionnaire  -  measuring  expertise,  trust  between  team  members,  and  general 
collaboration  opinions  among  members 


Ho:  no  significant  difference  between  face-to-face  and  asynchronous,  distributed 

collaboration  modes  on  the  collaboration  stages  and  cognitive  knowledge  building  processes 
in  the  NEO  collaborative  problem  domain 

Ho:  no  significant  difference  between  static  and  dynamic  knowledge  on  the  collaboration 

stages  and  cognitive  knowledge  building  processes  in  the  NEO  collaborative  problem  domain. 

Ho:  no  significant  interaction  between  collaboration  modes  and  knowledge  dynamics  on 
the  collaboration  stages  and  cognitive  knowledge  building  process 

Ho:  no  significant  difference  in  time  or  frequency  within  each  cognitive  process  state  across 
collaboration  mode  and  knowledge  dynamic  conditions 

Ho:  no  convergence  of  individual  mental  models  with  regard  to  collaboration  stages  and 
knowledge  building  processes 


Accomplishments 
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•  Development  of  a  Structural  Model  Of  Team  Collaboration 

>  Describes  team  collaboration  emphasizing  the  cognitive  aspects  of  the  collaboration  process. 

>  The  model  includes  the  domain  characteristics,  collaboration  stages,  meta-cognitive  processes,  information 

processing  tasks,  knowledge  required  for  each  information  processing  task  and  the  communication 
mechanisms  for  knowledge  building  and  information  processing  tasks. 

>  Starting  point  for  understanding  the  cognitive  mechanisms  of  team  collaboration 

>  Offers  a  model-based  approach  to  experimentation  of  team  collaboration 

>  Identifies  and  prioritizes  important  areas  of  research  in  team  collaboration 

>  Serves  as  a  design  guideline  for  an  agent-based  support  tool  for  team  collaboration. 

•  CASC  Phase  I  Collaboration  Experiment 

>  Examined  the  collaboration  stages  and  cognitive  processes  used  by  face-to-face  versus  asynchronous,  distributed 

teams  in  a  collaborative  problem  solving  task. 

•  Transition  of  Structural  Model  of  Team  Collaboration  to  Naval  Air  Systems  Command  Network 

Centric  Warfare  Simulation  Facility 

>  Collaboration  model  is  currently  being  transitioned  into  the  newly  developed  Naval  Air  Systems  Command 

Network  Centric  Warfare  Simulation  Facility  to  improve  collaboration  between  Commanders  both  at  the  front 
end  (e.g.  mission  planning),  and  during  mission  execution 

>  The  Naval  Air  Systems  Command  Network  Centric  Simulation  Facility  represents  connectivity  between  all  the 

relevant  assets  in  a  Network  Centric  battlegroup  including  air  (i.e.  Man-Flight  Simulation  Facility),  ships  (i.e. 
Aegis  Simulation  Facility),  Command  and  Control,  Battlespace  Visualization  (i.e.  Crew  Station  Technology 
Laboratory)  and  Collaboration  Technology  (i.e.  Collaboration  and  Automation  Research  Laboratory) 

>  The  collaboration  model  will  provide  a  significant  improvement  in  asynchronous,  distributed  collaborative  decision 

making  within  the  Network  Centric  Warfare  environment. 
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Journal  Articles 


•Warner,  N.W.,  Letsky,  M.,  Cowen,  M.(In  Press).  Structural  Model  of  Team  Collaboration.  Paper  being 
submitted  to  Journal  of  the  Human  Factors  and  Ergonomics  Society,  (peer  reviewed) 

•Warner,  N.W.,  Wroblewski,  E.,  Vanderwalker,  S.,  and  Verma,  N.  (In  Press).  The  Effect  of 
Collaboration  Mode  and  Knowledge  Distribution  on  the  Knowledge  Building  Process  in  an 
Asynchronous,  Distributed  Collaborative  Problem  Solving  Task.  Paper  being  submitted  to  the  Journal 
of  Human  Factors  and  Ergonomics  Society  (peer  reviewed) 

•  Warner,  N.  W.  and  Wroblewski,  E.(2003).  The  Cognitive  Processes  Used  in  Team  Collaboration 
During  Asynchronous,  Distributed  Decision  Making.  Paper  submitted  to  the  2004  Command  and 
Control  Research  and  Technology  Symposium,  June  15-17,  2004,  San  Diego,  CA.  (peer  reviewed) 

Technical  Reports 

•Warner,  N.W.,  Vanderwalker,  S.  and  Verma,  N.  (2003).  Collaborative  Knowledge  in  Asynchronous 
Collaboration  (CASC).  Collaboration  and  Knowledge  Management  Workshop  Proceedings,  January 
14-16,  2003.  Office  of  Naval  Research,  Code  342,  Arlington,  Va. 

•Warner,  N.W.  Vanderwalker,  S.,  Verma,  N.  and  Narkevicius,  J.  (2002).  State  of  the  Art  Review  of 
Human-Human  Collaboration  Research:  An  Integrated,  Multidisciplinary  Perspective.  Collaboration 
and  Knowledge  Management  Workshop  Proceedings,  January  14-16,  2003.  Office  of  Naval  Research, 
Code  342,  Arlington,  Va. 


Lessons  Learned 
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I  Both  face-to-face  and  asynchronous,  distributed  teams  demonstrated  behavior  that  supports  the 
existence  of  Team  Knowledge  Construction,  Collaborative  Problem  Solving  and 
Team  Consensus  stages  during  collaborative  problem  solving.  Phase  I  data  showed  that  the  stages 
are  task  dependent  as  the  Outcome  Evaluation  &  Revision  stage  was  not  used 

t  The  significant  cognitive  process  states  were  Understanding  the  Problem,  Individual  Knowledge 
and  Team  Knowledge  development.  Conventions  for  Transferring  Meaning,  and  developing 
Solution  Alternatives 

i  Knowledge  Distribution  (homogeneous  /  heterogeneous  information)  significantly  influenced  how 
team  knowledge  was  developed 

^  Asynchronous,  Distributed  teams  spent  more  time  in  Collaborative  Problem  Solving  stage 
than  face-to-face  teams.  Appears  to  be  more  difficult  to  solve  collaborative  problems  with  these 
types  of  teams  even  though  collaboration  environment  is  inherently  more  structured.  Need  to 
determine  why  these  teams  spend  more  time  so  collaborative  problem  solving  can  be  facilitated 

■  Face-to-face  teams  used  conventions  to  transfer  meaning  (e.g.  yellow  stickers,  maps)  whereas 
asynchronous,  distributed  teams  did  not  use  conventions.  Need  ways  to  easily  create  conventions 
with  asynchronous,  distributed  teams  (another  area  for  agent  support) 


